Channel bundling

ABSTRACT

A method and apparatus for processing message is described. In one embodiment, a channel layer is configured to form a channel with a corresponding process. The channel is associated with a group. A plurality of sockets is formed in a transport layer of a group communication protocol. Traffic of messages to and from the processes is distributed across the plurality of sockets in the transport layer.

TECHNICAL FIELD

Embodiments of the present invention relate to group communication, andmore specifically to processing of messages.

BACKGROUND

Group communication protocol designed for multicast communication may beused to communicate messages between endpoints forming a group.Communication endpoints can be processes or objects, or any entity thatcan send and receive messages to/from a group.

The destination address of a node may comprise an IP address and port ofa receiver socket. However, the use of such address provides somedisadvantages. For example, when a node is shunned and later rejoins,its physical address may have changed. A communication directed to a NICmay be interrupted if the NIC fails. Also, the sender's address may bechanged by a Network Address Translator. As such, it would be desirableto provide for a logical address scheme that resolves the aboveproblems. Further, it would be desirable to provide for a channelbundling scheme to distribute the traffic across a transport layer.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in which:

FIG. 1 illustrates a network architecture of a group communication inwhich embodiments of the present invention may be implemented.

FIG. 2 illustrates a block diagram of one embodiment of a structure of amessage.

FIG. 3 illustrates a block diagram of one embodiment of channel states.

FIG. 4 illustrates a flow diagram of one embodiment of a method forbundling channels for a group communication protocol.

FIG. 5 illustrates a block diagram of an exemplary computer system.

DETAILED DESCRIPTION

Described herein is a method and apparatus for channel bundling in agroup communication scheme. A channel is formed with a correspondingprocess, the channel associated with a group. A plurality of sockets isformed in a transport layer of a group communication protocol. Trafficto and from the processes is distributed across the plurality of socketsin the transport layer.

In the following description, numerous details are set forth. It will beapparent, however, to one skilled in the art, that the present inventionmay be practiced without these specific details. In some instances,well-known structures and devices are shown in block diagram form,rather than in detail, in order to avoid obscuring the presentinvention.

Some portions of the detailed descriptions which follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present invention also relates to apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions, and each coupledto a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein.

A machine-accessible storage medium includes any mechanism for storingor transmitting information in a form readable by a machine (e.g., acomputer). For example, a machine-accessible storage medium includesread only memory (“ROM”); random access memory (“RAM”); magnetic diskstorage media; optical storage media; flash memory devices; electrical,optical, acoustical or other form of propagated signals (e.g., carrierwaves, infrared signals, digital signals, etc.); etc.

Group Communication Architecture

FIG. 1 illustrates an exemplary network architecture of a groupcommunication 100, such as JGroups, in which embodiments of the presentinvention may operate.

JGroups is toolkit for reliable group communication. Processes can joina group, send messages to all members or single members and receivemessages from members in the group. The system keeps track of themembers in every group, and notifies group members when a new memberjoins, or an existing member leaves or crashes. A group is identified byits name. Groups do not have to be created explicitly; when a processjoins a non-existing group, that group will be created automatically.Member processes of a group can be located on the same host, within thesame LAN, or across a WAN. A member can be part of multiple groups.

The group communication architecture may comprise three parts: (1) achannel API 106 used by application programmers to build reliable groupcommunication applications, (2) building blocks 108, which are layeredon top of channel 106 and provide a higher abstraction level and (3) aprotocol stack 104, which implements the properties specified for agiven channel.

Channel 106 is connected to protocol stack 104. Whenever an applicationsends a message, channel 106 passes it on to protocol stack 104comprising several protocols 112, 114, 116, 118, 120. The topmostprotocol processes the message and the passes it on to the protocolbelow it. Thus, the message is handed from protocol to protocol untilthe bottom protocol puts it on the network 102. The same happens in thereverse direction: the bottom (transport) protocol listens for messageson network 102. When a message is received, it will be handed upprotocol stack 104 until it reaches channel 106. Channel 106 stores themessage in a queue until application 110 consumes it.

When an application 110 connects to a channel 106, protocol stack 106will be started, and when it disconnects protocol stack 104 will bestopped. When the channel 106 is closed, the stack 140 will bedestroyed, releasing its resources.

Channel

To join a group and send messages, a process has to create a channel andconnect to it using the group name (all channels with the same name forma group). The channel is the handle to the group. While connected, amember may send and receive messages to/from all other group members.The client leaves a group by disconnecting from the channel. A channelcan be reused: clients can connect to it again after havingdisconnected. However, a channel may allow only one client to beconnected at a time. If multiple groups are to be joined, multiplechannels can be created and connected to. A client signals that it nolonger wants to use a channel by closing it. After this operation, thechannel may not be used any longer.

Each channel has a unique address. Channels always know who the othermembers are in the same group: a list of member addresses can beretrieved from any channel. This list is called a view. A process canselect an address from this list and send a unicast message to it (alsoto itself), or it may send a multicast message to all members of thecurrent view. Whenever a process joins or leaves a group, or when acrashed process has been detected, a new view is sent to all remaininggroup members. When a member process is suspected of having crashed, asuspicion message is received by all non-faulty members. Thus, channelsreceive regular messages, view messages and suspicion messages. A clientmay choose to turn reception of views and suspicions on/off on a channelbasis.

Channels may be similar to BSD sockets: messages are stored in a channeluntil a client removes the next one (pull-principle). When no message iscurrently available, a client is blocked until the next availablemessage has been received.

A channel may be implemented over a number of alternatives for grouptransport. Therefore, a channel is an abstract class, and concreteimplementations are derived from it, e.g. a channel implementation usingits own protocol stack, or others using existing group transports suchas Jchannel and EnsChannel. Applications only deal with the abstractchannel class, and the actual implementation can be chosen at startuptime.

The properties for a channel may be specified in a colon-delimitedstring format. When creating a channel (JChannel) a protocol stack willbe created according to these properties. All messages will pass throughthis stack, ensuring the quality of service specified by the propertiesstring for a given channel.

Building Blocks

Channels are simple and primitive. They offer the bare functionality ofgroup communication, and have on purpose been designed after the simplemodel of BSD sockets, which are widely used and well understood. Thereason is that an application can make use of just this small subset ofJGroups, without having to include a whole set of sophisticated classes,that it may not even need. Also, a somewhat minimalistic interface issimple to understand: a client needs to know about 12 methods to be ableto create and use a channel (and oftentimes will only use 3-4 methodsfrequently).

Channels provide asynchronous message sending/reception, somewhatsimilar to UDP. A message sent is essentially put on the network and thesend( ) method will return immediately. Conceptual requests, orresponses to previous requests, are received in undefined order, and theapplication has to take care of matching responses with requests.

Also, an application has to actively retrieve messages from a channel(pull-style); it is not notified when a message has been received. Notethat pull-style message reception often needs another thread ofexecution, or some form of event-loop, in which a channel isperiodically polled for messages.

JGroups offers building blocks that provide more sophisticated APIs ontop of a Channel. Building blocks either create and use channelsinternally, or require an existing channel to be specified when creatinga building block. Applications communicate directly with the buildingblock, rather than the channel. Building blocks are intended to save theapplication programmer from having to write tedious and recurring code,e.g. request-response correlation.

Protocol Stack

As discussed above, JGroups provides two channel implementations: anEnsemble-based channel and its own channel based on a Java protocolstack. The latter is a protocol stack containing a number of protocollayers in a bidirectional list. FIG. 1 illustrates protocol stack 104with the following procotols: CAUSAL 112, GMS 114, MERGE 116, FRAG 118,UDP 120.

All messages sent and received over the channel have to pass through theprotocol stack. Every layer may modify, reorder, pass or drop a message,or add a header to a message. A fragmentation layer might break up amessage into several smaller messages, adding a header with an id toeach fragment, and re-assemble the fragments on the receiver's side.

The composition of the protocol stack, i.e. its layers, is determined bythe creator of the channel: a property string defines the layers to beused (and the parameters for each layer). This string might beinterpreted differently by each channel implementation; in JChannel itis used to create the stack, depending on the protocol names given inthe property.

Knowledge about the protocol stack is not necessary when only usingchannels in an application. However, when an application wishes toignore the default properties for a protocol stack, and configure theirown stack, then knowledge about what the individual layers are supposedto do is needed. Although it is syntactically possible to stack anylayer on top of each other (they all have the same interface), thiswouldn't make sense semantically in most cases.

Message

Data is sent between members in the form of messages. A message can besent by a member to a single member, or to all members of the group ofwhich the channel is an endpoint. An example of a structure of a message200 is illustrated in FIG. 2.

The message 200 may contain five fields: headers 202, destinationaddress 204, source address 206, flags 208, and payload 210.

A list of headers 202 can be attached to a message. Anything that shouldnot be in the payload 210 can be attached to message 200 as a header.Methods putHeader( ), getHeader( ), and removeHeader( ) of message 200can be used to manipulate headers 202.

The destination address 204 may include the address of the receiver. Ifnull, the message will be sent to all current group members.

The source address 206 may include the address of a sender. It can beleft null, and will be filled in by the transport protocol (e.g. UDP)before the message is put on the network 102.

One byte of the message 200 may be used for flags 208. Examples of flagsmay be OOB, LOW_PRIO and HIGH_PRIO.

The payload 210 may include the actual data (as a byte buffer). Themessage class contains convenience methods to set a serializable objectand to retrieve it again, using serialization to convert the objectto/from a byte buffer.

The message 200 may be similar to an IP packet and consists of thepayload (a byte buffer) and the addresses of the sender and receiver (asaddresses). Any message put on the network 102 can be routed to itsdestination (receiver address), and replies can be returned to thesender's address.

A message usually does not need to fill in the sender's address whensending a message; this is done automatically by the protocol stackbefore a message is put on the network. However, there may be cases,when the sender of a message wants to give an address different from itsown, so that for example, a response should be returned to some othermember.

The destination address (receiver) can be an Address, denoting theaddress of a member, determined e.g. from a message received previously,or it can be null, which means that the message will be sent to allmembers of the group. A typical multicast message, sending string“Hello” to all members would look like this:

Message msg=new Message(null, null, “Hello”.getBytes( ));channel.send(msg);

View

A View is a list of the current members of a group. It consists of aViewId, which uniquely identifies the view (see below), and a list ofmembers. Views are set in a channel automatically by the underlyingprotocol stack whenever a new member joins or an existing one leaves (orcrashes). All members of a group see the same sequence of views.

Note that there is a comparison function which orders all the members ofa group in the same way. Usually, the first member of the list is thecoordinator (the one who emits new views). Thus, whenever the membershipchanges, every member can determine the coordinator easily and withouthaving to contact other members.

The code below shows how to send a (unicast) message to the first memberof a view (error checking code omitted):

View myview=channel.getView( ); Address first=myview.getMembers().first( ); Message msg=new Message(first, null, “Hello world”);channel.send(msg);

Whenever an application is notified that a new view has been installed(e.g. by MembershipListener.viewAccepted( ) or Channel.receive( )), theview is already set in the channel. For example, callingChannel.getView( ) in a viewAccepted( ) callback would return the sameview (or possibly the next one in case there has already been a newview).

A ViewId is used to uniquely number views. It consists of the address ofthe view creator and a sequence number. ViewIds can be compared forequality and put in a hashtable as they implement equals( ) andhashCode( ) methods.

Whenever a group splits into subgroups, e.g. due to a network partition,and later the subgroups merge back together, a MergeView instead of aView will be received by the application. The MergeView class is asubclass of View and contains as additional instance variable the listof views that were merged. As an example if the group denoted by viewV1:(p,q,r,s,t) split into subgroups V2:(p,q,r) and V2:(s,t), the mergedview might be V3:(p,q,r,s,t). In this case the MergeView would containsa list of 2 views: V2:(p,q,r) and V2:(s,t).

Channel States

A state transition diagram 300 for the major states a channel can assumeare shown in FIG. 3. In order to join a group and send messages, aprocess has to create a channel. A channel is like a socket. When aclient connects to a channel, it gives the name of the group it wouldlike to join. Thus, a channel is (in its connected state) alwaysassociated with a particular group. The protocol stack takes care thatchannels with the same group name find each other: whenever a clientconnects to a channel given group name G, then it tries to find existingchannels with the same name, and joins them, resulting in a new viewbeing installed (which contains the new member). If no members exist, anew group will be created.

When a channel is first created at 308, it is in the unconnected state302. An attempt to perform certain operations which are only valid inthe connected state (e.g. send/receive messages) will result in anexception. After a successful connection by a client, it moves to theconnected state 304. Now channels will receive messages, views andsuspicions from other members and may send messages to other members orto the group. Getting the local address of a channel is guaranteed to bea valid operation in this state (see below). When the channel isdisconnected, it moves back to the unconnected state 302. Both aconnected and unconnected channel may be closed 306, which makes thechannel unusable for further operations. Any attempt to do so willresult in an exception. When a channel is closed directly from aconnected state, it will first be disconnected, and then closed.

Creating a Channel

A channel can be created in two ways: an instance of a subclass ofChannel is created directly using its public constructor (e.g. newJChannel( )), or a channel factory is created, which—uponrequest—creates instances of channels. We will only look at the firstmethod of creating channel: by direct instantiation. Note thatinstantiation may differ between the various channel implementations. Asexample we will look at JChannel.

The public constructor of JChannel looks as follows:

-   -   public JChannel(Object properties) throws ChannelException { }

It creates an instance of JChannel. The properties argument defines thecomposition of the protocol stack (number and type of layers, parametersfor each layer, and their order). For JChannel, this has to be a String(see Section 3.7.2, “Channel properties” for how to define one's ownproperties string). An example of a channel creation is:

String props=“UDP(mcast_addr=228.1.2.3;mcast_port=45566;ip_tt1=32):” +“PING(timeout=3000;num_initial_members=6):” + “FD(timeout=5000):” +“VERIFY_SUSPECT(timeout=1500):” +“pbcast.STABLE(desired_avg_gossip=10000):” +“pbcast.NAKACK(gc_lag=10;retransmit_timeout=3000):” +“UNICAST(timeout=5000;min_wait_time=2000):” + “FRAG:” +“pbcast.GMS(initial_mbrs_timeout=4000;join_timeout=5000;” +“join_retry_timeout=2000;shun=false;print_local_addr=false)”; JChannelchannel; try { channel=new JChannel(props); } catch(Exception ex) { //channel creation failed }

The argument is a colon-delimited string of protocols, specified frombottom to top (left to right). The example properties argument will beused to create a protocol stack that uses IP Multicast (UDP) as bottomprotocol, the PING protocol to locate the initial members, FD forfailure detection, VERIFY_SUSPECT for double-checking of suspectedmembers, STABLE for garbage collection of messages received by allmembers, NAKACK for lossless delivery of multicast messages, UNICAST forlossless delivery of unicast messages and GMS for group membership(handling of join or leave requests).

If the properties argument is null, the default properties will be used.An exception will be thrown if the channel cannot be created. Possiblecauses include protocols that were specified in the property argument,but were not found, or wrong parameters to protocols.

Using XML to Define a Protocol Stack

In version 2.0 of JGroups an XML-based scheme to define protocol stackswas introduced. Instead of specifying a string containing the protocolspec, an URL pointing to a valid protocol stack definition can be given.For example, the Draw demo can be launched as follows:

-   -   java org.javagroups.demos.Draw-props        file:/home/bela/vsync.xml

or

-   -   java org.javagroups.demos.Draw-props        http://www.jgroups.com/udp.xml

In the latter case, an application downloads its protocol stackspecification from a server, which allows for central administration ofapplication properties. Plain and XML-based configuration of protocolstacks will be discussed in more detail in chapter.

Channel Properties

A property string consists of a number of properties separated bycolons:

“<prop1>(arg1=val1):<prop2>(arg1=val1;arg2=val2):<prop3>:<propn>”

Each property relates directly to a protocol layer, which is implementedas a Java class. When a protocol stack is to be created based on theabove property string, the first property becomes the bottom-most layer,the second one will be placed on the first, etc: the stack is createdfrom the bottom to the top, as the string is parsed from left to right.Each property has to be the name of a Java class that resides in theorg.jgroups.stack.protocols package. Note that only the base name has tobe given, not the fully specified class name (UDP instead oforg.jgroups.stack.protocols.UDP). If the protocol class is not found,JGroups assumes that the name given is a fully qualified classname andwill therefore try to instantiate that class. If this does not work anexception is thrown. This allows for protocol classes to reside indifferent packages altogether, e.g. a valid protocol name could becom.sun.eng.protocols.reliable.UCAST.

Each layer may have zero or more arguments, which are specified as alist of name/value pairs in parentheses directly after the property. Inthe example above, the first protocol layer has 1 argument, the second2, the third none. When a layer is created these properties (if thereare any) will be set in a protocol, thus configuring the protocol stackaccording to the channel creator.

As an example the property string below instructs JGroups to create aJChannel with protocols UDP, PING, FD and GMS:

“UDP(mcast_addr=228.10.9.8;mcast_port=5678):PING:FD:GMS”

The UDP protocol layer is at the bottom of the stack, and it should usemcast address 228.10.9.8. and port 5678 rather than the default IPmulticast address and port. Property UDP refers to a classorg.jgroups.stack.protocols.UDP, which is subsequently loaded and aninstance of which is created as protocol layer. If any of these classesare not found, an exception will be thrown and the construction of thestack will be aborted.

Note that all members in a group have to have the same protocol stack.

Setting Options

A number of options can be set in a channel. To do so, the followingmethod is used:

-   -   public void setOpt(int option, Object value);

Arguments are the options number and a value. The following options arecurrently recognized:

Channel.BLOCK

The argument is a boolean object. If true, block messages will bereceived. If this option is set to true, views will also be set to true.Default is false.

Channel.LOCAL

Local delivery. The argument is a boolean value. If set to true, amember will receive all messages it sent to itself. Otherwise, allmessages sent by itself will be discarded. This option allows to sendmessages to the group, without receiving a copy. Default is true(members will receive their own copy of messages multicast to thegroup).

Channel.AUTO_RECONNECT

When set to true, a shunned channel will leave the group and then try toautomatically re-join. Default is false

Channel.AUTO_GETSTATE

When set to true a shunned channel, after reconnection, will attempt tofetch the state from the coordinator. This requires AUTO_RECONNECT to betrue as well. Default is false.

The equivalent method to get options is getOpt( ):

-   -   public Object getOpt(int option);

Given an option, the current value of the option is returned.

Connecting to a Channel

When a client wants to join a group, it connects to a channel giving thename of the group to be joined:

-   -   public void connect(String groupname) throws ChannelClosed;

The group address is a string, naming the group to be joined. Allchannels that are connected to the same group (same name) form a group.Messages multicast on any channel in the group will be received by allmembers (including the one who sent it [3]).

The method returns as soon as the group has been joined successfully. Ifthe channel is in the closed state (see FIG. 3), an exception will bethrown. If there are no other members, i.e. no other client hasconnected to a group with this name, then a new group is created and themember joined. The first member of a group becomes its coordinator. Acoordinator is in charge of multicasting new views whenever themembership changes.

Getting the Local Address and the Group Name

Method getLocalAddress( ) returns the local address of the channel. Inthe case of JChannel, the local address is generated by the bottom-mostlayer of the protocol stack when the stack is connected to. That meansthat—depending on the channel implementation—the local address may ormay not be available when a channel is in the unconnected state.

-   -   public Address getLocalAddress( );

Method getChannelName( ) returns the name of the group in which thechannel is a member:

-   -   public String getChannelName( );

Again, the result is undefined if the channel is in the unconnected orclosed state.

Getting the Current View

The following method can be used to get the current view of a channel:

-   -   public View getview( );

This method does not retrieve a new view (message) from the channel, butonly returns the current view of the channel. The current view isupdated every time a view message is received: when method receives iscalled, and the return value is a view, before the view is returned, itwill be installed in the channel, i.e. it will become the current view.

Calling this method on an unconnected or closed channel isimplementation defined. A channel may return null, or it may return thelast view it knew of.

Sending a Message

Once the channel is connected, messages can be sent using the send( )methods:

-   -   public void send(Message msg) throws ChannelNotConnected,        ChannelClosed;    -   public void send(Address dst, Address src, Object obj)    -   throws ChannelNotConnected, ChannelClosed;

The first send( ) method has only one argument, which is the message tobe sent. The message's destination should either be the address of thereceiver (unicast) or null (multicast). When it is null, the messagewill be sent to all members of the group (including itself). The sourceaddress may be null; if it is, it will be set to the channel's address(so that recipients may generate a response and send it back to thesender).

The second send( ) method is a helper method and uses the former methodinternally. It requires the address of receiver and sender and an object(which has to be serializable), constructs a Message and sends it.

If the channel is not connected, or was closed, an exception will bethrown upon attempting to send a message.

Here's an example of sending a (multicast) message to all members of agroup:

Hashtable data; // any serializable data try { channel.send(null, null,data); } catch(Exception ex) { // handle errors }

The null value as destination address means that the message will besent to all members in the group. The sender's address will be filled inby the bottom-most protocol. The payload is a hashtable, which will beserialized into the message's buffer and unserialized at the receiver'send. Alternatively, any other means of generating a byte buffer andsetting the message's buffer to it (e.g. using Message.setBuffer( ))would also work.

Here's an example of sending a (unicast) message to the first member(coordinator) of a group:

Address receiver; Message msg; Hashtable data; try {receiver=channel.getView( ).getMembers( ).first( );channel.send(receiver, null, data); } catch(Exception ex) { // handleerrors }

It creates a Message with a specific address for the receiver (the firstmember of the group). Again, the sender's address can be left null as itwill be filled in by the bottom-most protocol.

Receiving a Message

Method receives is used to receive messages, views, suspicions andblocks:

-   -   public Object receive(long timeout)    -   throws ChannelNotConnected, ChannelClosed, Timeout;

A channel receives messages asynchronously from the network and storesthem in a queue. When receives is called, the next available messagefrom the top of that queue is removed and returned. When there are nomessages on the queue, the method will block. If timeout is greater than0, it will wait the specified number of milliseconds for a message to bereceived, and throw a TimeoutException exception if none was receivedduring that time. If the timeout is 0 or negative, the method will waitindefinitely for the next available message.

Depending on the channel options (see Section 3.7.3, “Setting options”),the following types of objects may be received:

Message

A regular message. To send a response to the sender, a new message canbe created. Its destination address would be the received message'ssource address. Method Message.makeReply( ) is a helper method to createa response.

View

A view change, signalling that a member has joined, left or crashed. Theapplication may or may not perform some action upon receiving a viewchange (e.g. updating a GUI object of the membership, or redistributinga load-balanced collaborative task to all members). Note that a longeraction, or any action that blocks should be performed in a separatethread. A MergeView will be received when 2 or more subgroups mergedinto one (see Section 3.5.2, “MergeView” for details). Here, a possiblestate merge by the application needs to be done in a separate thread.

SuspectEvent

Notification of a member that is suspected. MethodSuspectEvent.getMember( ) retrieves the address of the suspected member.Usually this message will be followed by a view change.

BlockEvent

The application has to stop sending messages. When the application hasstopped sending messages, it needs to acknowledge this message with aChannel.blockOk( ) method.

The BlockEvent reception can be used to complete pending tasks, e.g.send pending messages, but once Channel.blockOk( ) has been called, allthreads that send messages (calling Channel.send( ) or Channel.down( ))will be blocked until FLUSH unblocks them.

UnblockEvent

The application can resume sending messages. Any previously messagesblocked by FLUSH will be unblocked; when the UnblockEvent is receivedthe channel has already been unblocked.

GetStateEvent

Received when the application's current state should be saved (for alater state transfer. A copy of the current state should be made(possibly wrapped in a synchronized statement and returned callingmethod Channel.returnState( ). If state transfer events are not enabledon the channel (default), then this event will never be received. Thismessage will only be received with the Virtual Synchrony suite ofprotocols (see the Programmer's Guide).

StreamingGetStateEvent

Received when the application's current state should be provided to astate requesting group member. If state transfer events are not enabledon the channel (default), or if channel is not configured withpbcast.STREAMING_STATE_TRANSFER then this event will never be received.

SetStateEvent

Received as response to a getState(s) method call. The argument containsthe state of a single member (byte[ ]) or of all members (Vector). Sincethe state of a single member could also be a vector, the interpretationof the argument is left to the application.

StreamingSetStateEvent

Received at state requesting member when the state InputStream becomesready for reading. If state transfer events are not enabled on thechannel (default), or if channel is not configured withpbcast.STREAMING_STATE_TRANSFER then this event will never be received.

The caller has to check the type of the object returned. This can bedone using the instance of operator, as follows:

Object obj; Message msg; View v; obj=channel.receive(0); // wait foreverif(obj instanceof Message) msg=(Message)obj; else if(obj instanceofView) v=(View)obj; else ; // don't handle suspicions or blocks

If for example views, suspicions and blocks are disabled, then thecaller is guaranteed to only receive return values of type Message. Inthis case, the return value can be cast to a Message directly, withoutusing the instance of operator.

If the channel is not connected, or was closed, a correspondingexception will be thrown.

The example below shows how to retrieve the “Hello world” string from amessage:

Message msg; // received above String s; try { s=(String)msg.getObject(); // error if object not Serializable // alternative: s=newString(msg.getBuffer( )); } catch(Exception ex) { // handle errors, e.g.casting error above) }

The Message.getObject( ) method retrieves the message's byte buffer,converts it into a (serializable) object and returns the object.

Channel Bundling

Traffic can be distributed across multiple sockets in a transport. Forexample, if we have eth0 and eth1, and both are 100 Mbps “full duplex”.Then, we should be able to send 1000 Mbps on eth0 and 100 Mbps on eth1simultaneously.

FIG. 4 illustrates a flow diagram of one embodiment for a method forbundling channels. At 402, a channel is formed with a correspondingprocess. The channel is associated with a group. At 404, a plurality ofsockets is formed in a transport layer of a group communicationprotocol. At 406, traffic of messages to and from the processes isdistributed across the plurality of sockets in the transport layer.

Logical Addresses

Conventionally, the address chosen by each node is essentially the IPaddress and port of the receiver socket. However, for the followingreasons, this is not good enough:

In the case where the node is shunned (excluded) and re-joins afterleaving, the same logical address would be desirable although thephysical address has changed.

In the case where a NIC goes down, we want to continue sending/receivingon a different NIC.

In the case where the sender sends on all available NICs(send_on_all_interfaces=“true”), this means that—if we take thereceiver's datagram packet's address to be the identity of the sender—weget N different identities; 1 for each interface the message is sent on.

In the case where a Network Address Translation is used, the sender'saddress might get changed by the NAT.

In order to overcome the above challenges, one embodiment of a logicaladdress may be as follows. A logical address may be selected, either byJGroups, or set by a user on channel creation. The lifetime of thisaddress is the lifetime of the process in which the channel is created,or until channel.close( ) or disconnects is called.

Each member may have a small cache, in which it associates the logicaladdresses for messages received with the sender's address. When amessage is to be sent to a logical address (unicast message), thecorresponding physical address is looked up from the cache. Inaccordance with another embodiment, there may be multiple physicaladdresses if the same message was sent on different interfaces(send_on_all_interfaces=“true”).

The logical addresses must be picked such that they cannot be reusedafter disconnect( )/close( ). FIG. 4 illustrates a table of logicaladdresses in accordance with one embodiment.

The logical address scheme may be written in Java to provide reliablemulticast communication. First, a logical address is formed upon acreation of a channel of a group communication protocol. An associationof the logical address with a message received with a sender's addressis stored in a cache of a member of the group. The logical address isdissolved when the channel is to be closed or disconnected. The logicaladdress is formed independently of an IP address and a port of areceiver socket. The logical address may be selected by a user onchannel creation. Each channel may be associated with a correspondinggroup. The logical address is selected such that it cannot be reusedafter a corresponding channel is closed or disconnected.

Computer System

FIG. 5 illustrates a diagrammatic representation of a machine in theexemplary form of a computer system 500 within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein, may be executed. In alternativeembodiments, the machine may be connected (e.g., networked) to othermachines in a LAN, an intranet, an extranet, or the Internet. Themachine may operate in the capacity of a server or a client machine inclient-server network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine may be apersonal computer (PC), a tablet PC, a set-top box (STB), a PersonalDigital Assistant (PDA), a cellular telephone, a web appliance, aserver, a network router, switch or bridge, or any machine capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein.

The exemplary computer system 500 includes a processing device 502, amain memory 504 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM) or RambusDRAM (RDRAM), etc.), a static memory 506 (e.g., flash memory, staticrandom access memory (SRAM), etc.), and a data storage device 518, whichcommunicate with each other via a bus 530.

Processing device 502 represents one or more general-purpose processingdevices such as a microprocessor, central processing unit, or the like.More particularly, the processing device may be complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,or processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processing device 502may also be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. The processing device 502 is configured to execute theprocessing logic 526 for performing the operations and steps discussedherein.

The computer system 500 may further include a network interface device508. The computer system 500 also may include a video display unit 510(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 512 (e.g., a keyboard), a cursor controldevice 514 (e.g., a mouse), and a signal generation device 516 (e.g., aspeaker).

The data storage device 518 may include a machine-accessible storagemedium 530 on which is stored one or more sets of instructions (e.g.,software 522) embodying any one or more of the methodologies orfunctions described herein. The software 522 may also reside, completelyor at least partially, within the main memory 504 and/or within theprocessing device 502 during execution thereof by the computer system500, the main memory 504 and the processing device 502 also constitutingmachine-accessible storage media. The software 522 may further betransmitted or received over a network 520 via the network interfacedevice 508.

The machine-accessible storage medium 530 may also be used to JGroupsand concurrent stack configurations 524. JGroups and concurrent stackconfigurations 524 may also be stored in other sections of computersystem 500, such as static memory 506.

While the machine-accessible storage medium 530 is shown in an exemplaryembodiment to be a single medium, the term “machine-accessible storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of instructions. The term“machine-accessible storage medium” shall also be taken to include anymedium that is capable of storing, encoding or carrying a set ofinstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the present invention.The term “machine-accessible storage medium” shall accordingly be takento include, but not be limited to, solid-state memories, optical andmagnetic media, and carrier wave signals.

Thus, a method and apparatus for selecting and configuring a logicaladdress has been described. It is to be understood that the abovedescription is intended to be illustrative, and not restrictive. Manyother embodiments will be apparent to those of skill in the art uponreading and understanding the above description. The scope of theinvention should, therefore, be determined with reference to theappended claims, along with the full scope of equivalents to which suchclaims are entitled.

1. A computer-implemented method for processing messages, the methodcomprising: forming a channel with a corresponding process, the channelassociated with a group; forming a plurality of sockets in a transportlayer of a group communication protocol; and distributing a traffic ofmessages to and from the processes across the plurality of sockets inthe transport layer.
 2. The method of claim 1 further comprising:receiving packets from a plurality of senders with the transport layer,the transport layer comprising a multicast receiver thread, a unicastreceiver thread, and a connection table.
 3. The method of claim 2further comprising: processing a packet with a thread from a threadpool.
 4. The method of claim 3 further comprising: forming anotherthread to process the packet when all threads from the thread pool arebusy.
 5. The method of claim 3 wherein the thread sends the packets upthe transport layer to a corresponding channel of a channel layer, abuilding block layered on top of the channel layer, and an applicationprogramming interface layered on top of the building block.
 6. Themethod of claim 1 further comprising: forming a logical address upon acreation of a channel of the group communication protocol; storing anassociation of the logical address with a message received with asender's address in a cache of a member of the group; and dissolving thelogical address when the channel is to be closed or disconnected.
 7. Themethod of claim 6 wherein the logical address is formed independently ofan IP address and a port of a receiver socket.
 8. The method of claim 6wherein the logical address is selected by a user on channel creation.9. The method of claim 6 wherein the logical address is selected suchthat it cannot be reused after a corresponding channel is closed ordisconnected.
 10. An article of manufacture comprising: amachine-accessible storage medium including data that, when accessed bya machine, cause the machine to perform a method comprising: forming achannel with a corresponding process, the channel associated with agroup; forming a plurality of sockets in a transport layer of a groupcommunication protocol; and distributing a traffic of messages to andfrom the processes across the plurality of sockets in the transportlayer.
 11. The article of manufacture of claim 10 wherein the methodfurther comprises: receiving packets from a plurality of senders withthe transport layer, the transport layer comprising a multicast receiverthread, a unicast receiver thread, and a connection table.
 12. Thearticle of manufacture of claim 11 wherein the method further comprises:processing a packet with a thread from a thread pool.
 13. The article ofmanufacture of claim 12 wherein the method further comprises: forminganother thread to process the packet when all threads from the threadpool are busy.
 14. The article of manufacture of claim 12 wherein thethread sends the packets up the transport layer to a correspondingchannel of a channel layer, a building block layered on top of thechannel layer, and an application programming interface layered on topof the building block.
 15. The article of manufacture of claim 10wherein the method further comprises: forming a logical address upon acreation of a channel of a group communication protocol; storing anassociation of the logical address with a message received with asender's address in a cache of a member of the group; and dissolving thelogical address when the channel is to be closed or disconnected. 16.The article of manufacture of claim 15 wherein the logical address isformed independently of an IP address and a port of a receiver socket.17. The article of manufacture of claim 15 wherein the logical addressis selected by a user on channel creation.
 18. The article ofmanufacture of claim 15 wherein the logical address is selected suchthat it cannot be reused after a corresponding channel is closed ordisconnected.
 19. An apparatus for processing messages comprising: anapplication programming interface for receiving and sending messages; abuilding block layer coupled to the application programming interface; achannel layer coupled to the building block layer; and a transportprotocol stack coupled to the channel layer for implementing propertiesspecified by the channel layer, wherein the channel layer is configuredto form a channel with a corresponding process, the channel associatedwith a group, to form a plurality of sockets in a transport layer of agroup communication protocol, and to distribute a traffic of messages toand from the processes across the plurality of sockets in the transportlayer.
 20. The apparatus of claim 19 wherein the channel layer is tofurther receive packets from a plurality of senders with the transportlayer, the transport layer comprising a multicast receiver thread, aunicast receiver thread, and a connection table.
 21. The apparatus ofclaim 19 wherein the channel layer is to further process a packet with athread from a thread pool.
 22. The apparatus of claim 19 wherein thechannel layer is to further form another thread to process the packetwhen all threads from the thread pool are busy.
 23. The apparatus ofclaim 19 wherein the thread sends the packets up the transport layer toa corresponding channel of a channel layer, a building block layered ontop of the channel layer, and an application programming interfacelayered on top of the building block.
 24. The apparatus of claim 19wherein the channel layer is to further form a logical address upon acreation of a channel of the group communication protocol, to store anassociation of the logical address with a message received with asender's address in a cache of a member of the group, and to dissolvethe logical address when the channel is to be closed or disconnected.25. The apparatus of claim 19 wherein the logical address is formedindependently of an IP address and a port of a receiver socket.
 26. Theapparatus of claim 19 wherein the logical address is selected by a useron channel creation.
 27. The method of claim 19 wherein the logicaladdress is selected such that it cannot be reused after a correspondingchannel is closed or disconnected.